Ai governance using tamper proof model metrics

ABSTRACT

One example of a system comprises using a processor for identifying a model to be validated that is stored in a repository; automatically computing and recording one or more model metrics for the model to be validated in a tamper-proof manner; comparing the computed tamper-proof metrics with one or more encoded rules and policies to determine if the model to be validated complies with the one or more encoded rules and policies; and outputting a notification to a device indicating a validation status of the model to be validated based on the comparison of the computed tamper-proof metrics with the one or more encoded rules and policies.

BACKGROUND

In regulated industries such as banking and insurance, any kind of modelneeds to undergo proper validation before it can be deployed toproduction. Typically, there are model validators who perform thevalidation of the models and approve or reject them. Data scientistsspend a lot of time building the model—which is in the order of months.One problem that enterprises have is that often data scientists build amodel without worrying about governance rules and policies. Hence, whenthe model validator validates the model, the model is rejected and thenthe data scientists have to go back to start over to fix the model. Thisdelays the deployment of models to production.

SUMMARY

Aspects of the disclosure may include a computer-implemented method,computer program product, and system. One example of the methodcomprises identifying a model to be validated that is stored in arepository; automatically computing and recording one or more modelmetrics for the model to be validated in a tamper-proof manner;comparing the computed tamper-proof metrics with one or more encodedrules and policies to determine if the model to be validated complieswith the one or more encoded rules and policies; and outputting anotification to a device indicating a validation status of the model tobe validated based on the comparison of the computed tamper-proofmetrics with the one or more encoded rules and policies.

The above summary is not intended to describe each illustratedembodiment or every implementation of the present disclosure.

DRAWINGS

Understanding that the drawings depict only exemplary embodiments andare not therefore to be considered limiting in scope, the exemplaryembodiments will be described with additional specificity and detailthrough the use of the accompanying drawings, in which:

FIG. 1 is a high-level block diagram of one embodiment of an examplesystem AI governance system.

FIG. 2 is a block diagram depicting one embodiment of an example AIvalidator.

FIG. 3 is a flow chart depicting one embodiment of an example method ofenforcing governance policy using tamper proof model metrics.

FIG. 4 depicts one embodiment of a cloud computing environment.

FIG. 5 depicts one embodiment of abstraction model layers.

In accordance with common practice, the various described features arenot drawn to scale but are drawn to emphasize specific features relevantto the exemplary embodiments.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings that form a part hereof, and in which is shown byway of illustration specific illustrative embodiments. However, it is tobe understood that other embodiments may be utilized and that logical,mechanical, and electrical changes may be made. Furthermore, the methodpresented in the drawing figures and the specification is not to beconstrued as limiting the order in which the individual steps may beperformed. The following detailed description is, therefore, not to betaken in a limiting sense.

As used herein, “a number of” when used with reference items, means oneor more items. For example, “a number of different types of networks” isone or more different types of networks.

Further, the phrases “at least one”, “one or more,” and “and/or” areopen-ended expressions that are both conjunctive and disjunctive inoperation. For example, each of the expressions “at least one of A, Band C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “oneor more of A, B, or C” and “A, B, and/or C” means A alone, B alone, Calone, A and B together, A and C together, B and C together, or A, B andC together. In other words, “at least one of”, “one or more of”, and“and/or” mean any combination of items and number of items may be usedfrom the list, but not all of the items in the list are required. Theitem may be a particular object, a thing, or a category. Additionally,the amount or number of each item in a combination of the listed itemsneed not be the same. For example, in some illustrative examples, “atleast one of A, B, and C” may be, for example, without limitation, twoof item A; one of item B; and ten of item C; or 0 of item A; four ofitem B and seven of item C; or other suitable combinations.

Additionally, the term “a” or “an” entity refers to one or more of thatentity. As such, the terms “a” (or “an”), “one or more” and “at leastone” can be used interchangeably herein. It is also to be noted that theterms “comprising,” “including,” and “having” can be usedinterchangeably.

Furthermore, the term “automatic” and variations thereof, as usedherein, refers to any process or operation done without material humaninput when the process or operation is performed. However, a process oroperation can be automatic, even though performance of the process oroperation uses material or immaterial human input, if the input isreceived before performance of the process or operation. Human input isdeemed to be material if such input influences how the process oroperation will be performed. Human input that consents to theperformance of the process or operation is not deemed to be “material.”

In the present disclosure, a system and method are proposed to reducethe likelihood of models getting rejected by human validators whichdelays the model development lifecycle. In particular, a mechanism isproposed to validate the model automatically against artificialintelligence (AI) governance rules and policies in order to provide anadvance warning of the model metrics which need to be fixed so that themodel can pass the model validation process. The embodiments describedherein include various features such as encoding of AI governance rulesand policies in a policy engine; tamper proof capturing of model metricsduring the model development lifecycle along with lineage; and automatedvalidation of the AI Governance policies and rules when a model is savedin a model repository.

FIG. 1 is a high-level block diagram of one embodiment of an example AIgovernance system 100. The system 100 includes one or more clientdevices 102-1 . . . 102-N (herein referred to collectively as clientdevices 102), a repository 106, and a control unit 108 (also referred toherein as AI validator 108) coupled together via a network 110. Thenetwork 110 can be implemented using any number of any suitable physicaland/or logical communications topologies. The network 110 may includeone or more private or public computing networks. For example, network110 may comprise a private network. Alternatively, or additionally,network 110 may comprise a public network, such as the Internet. Thus,network 110 may form part of a packet-based network, such as a localarea network, a wide-area network, and/or a global network such as theInternet. Network 110 can include one or more servers, networks, ordatabases, and can use one or more communication protocols to transferdata between the repository 106, the control unit 108, and the clientdevices 102.

Furthermore, although illustrated in FIG. 1 as a single entity, in otherexamples network 110 may comprise a plurality of networks, such as acombination of public and/or private networks. The communicationsnetwork 110 can include a variety of types of physical communicationchannels or “links.” The links can be wired, wireless, optical, and/orany other suitable media. In addition, the communications network 110can include a variety of network hardware and software for performingrouting, switching, and other functions, such as routers, switches, basestations, bridges or any other equipment that may be useful tofacilitate communicating data. Furthermore, it is to be understood thatdifferent devices in the system 100 can utilize different networks. Forexample, in some embodiments, some client devices 102 can becommunicatively coupled to the control unit 108 via a cellular networkwhile the repository 106 and/or other client devices 102 arecommunicatively coupled to the control unit 108 via a private wide areanetwork or the internet.

Additionally, it is to be understood that although control unit 108 isdepicted as a separate device communicatively coupled to the clientdevices 102 and repository 106 via the network 110, the control unit 108can be implemented differently in other embodiments. For example, insome embodiments, the control unit 108 is implemented as part of theclient devices 102. That is, the control unit 108 can be implementedusing computing resources of the client devices 102, such as processingunit, memory, etc. In other embodiments, some functions described hereinas being performed by the control unit 108 can be performed in one ormore of the client devices 102 while other functions are performed by aseparate device communicatively coupled to the client devices 102. Thus,the functionality described herein as being performed by the controlunit 108 can be distributed across two or more devices to enable thefunctionality of the control unit 108 discussed herein. That is, one ormore processors, interfaces, memories, etc. in each of two or moredevices can be configured to implement the functionality of the controlunit 108. For example, in some embodiments, one or more processes/agentscan be executed on each client device 102 to capture content to beanalyzed which is then communicated to the control unit 108 for furtheranalysis. Thus, it is to be understood that the control unit 108 can beimplemented differently in various embodiments.

Each of the client devices 102 can be implemented as a mobile device(such as, but not limited to, a smart phone, tablet, wearable device,augmented reality (AR)/virtual reality (VR) device, etc.), desktopcomputer, or laptop computer, etc. In addition, although repository 106is depicted in FIG. 1 as a single device storing both rules and policies112 as well as models 114, it is to be understood that repository 106can be implemented differently in other embodiments. For example, insome embodiments, the repository stores only the models 114, whereas therules and policies 112 are stored in the control unit 108. Additionally,the repository 106 and/or control unit 108 can be implemented within acloud computer system or using one or more cloud computing services.Consistent with various embodiments, a cloud computer system can includea network-based, distributed data processing system that provides one ormore cloud computing services. In certain embodiments, a cloud computersystem can include many computers, hundreds or thousands of them,disposed within one or more data centers and configured to shareresources over the network. However, it is to be understood that cloudcomputer systems are not limited to those which include hundreds orthousands of computers and can include few than hundreds of computers.

The control unit 108 is configured to implement an AI governance policyengine. For example, the control unit 108 can be configured to implementmachine learning techniques to evaluate a model against the rules andpolicies 112 which are encoded into the AI governance policy engine.Example machine learning techniques can include can comprise algorithmsor models that are generated by performing supervised, unsupervised, orsemi-supervised training on a dataset. Machine learning algorithms caninclude, but are not limited to, decision tree learning, associationrule learning, artificial neural networks, deep learning, inductivelogic programming, support vector machines, clustering, Bayesiannetworks, reinforcement learning, representation learning,similarity/metric training, sparse dictionary learning, geneticalgorithms, rule-based learning, and/or other machine learningtechniques.

For example, the machine learning algorithms can utilize one or more ofthe following example techniques: K-nearest neighbor (KNN), learningvector quantization (LVQ), self-organizing map (SOM), logisticregression, ordinary least squares regression (OLSR), linear regression,stepwise regression, multivariate adaptive regression spline (MARS),ridge regression, least absolute shrinkage and selection operator(LASSO), elastic net, least-angle regression (LARS), probabilisticclassifier, naïve Bayes classifier, binary classifier, linearclassifier, hierarchical classifier, canonical correlation analysis(CCA), factor analysis, independent component analysis (ICA), lineardiscriminant analysis (LDA), multidimensional scaling (MDS),non-negative metric factorization (NMF), partial least squaresregression (PLSR), principal component analysis (PCA), principalcomponent regression (PCR), Sammon mapping, t-distributed stochasticneighbor embedding (t-SNE), bootstrap aggregating, ensemble averaging,gradient boosted decision tree (GBRT), gradient boosting machine (GBM),inductive bias algorithms, Q-learning, state-action-reward-state-action(SARSA), temporal difference (TD) learning, apriori algorithms,equivalence class transformation (ECLAT) algorithms, Gaussian processregression, gene expression programming, group method of data handling(GMDH), inductive logic programming, instance-based learning, logisticmodel trees, information fuzzy networks (IFN), hidden Markov models,Gaussian naïve Bayes, multinomial naïve Bayes, averaged one-dependenceestimators (AODE), Bayesian network (BN), classification and regressiontree (CART), chi-squared automatic interaction detection (CHAID),expectation-maximization algorithm, feedforward neural networks, logiclearning machine, self-organizing map, single-linkage clustering, fuzzyclustering, hierarchical clustering, Boltzmann machines, convolutionalneural networks, recurrent neural networks, hierarchical temporal memory(HTM), and/or other machine learning techniques.

The AI governance policy engine enables a user, such as the Chief RiskOfficer of an organization, to define a set of AI Governance rules andpolicies that are to be enforced in the organization. Typically, theseare the same rules which are validated by human model validators. Somesample rules can include, but are not limited to, a rule that an AImodel should only be built using governed data. As used herein, governeddata is data that has been through cleansing process and complies with aselected quality standard. In other words, governed data is data thathas undergone specified quality checks. Governed data can be stored ingoverned catalogs such that data used from the governed catalogs isensured to have gone through the specified quality checks. Non-governeddata is typically easily accessible. For example, a data scientist canmake use of data which is easily available on their laptop or a commonmachine. However, such data is not necessarily governed data and, thus,poses quality concerns. Hence, in some embodiments, the system describedherein ensures that models are built only using governed data which isavailable in a governed catalog and has undergone proper data qualitychecks.

Other example rules include a rule that an AI model should have fairnessmetric above a given threshold, e.g. 80%, and a rule that an AI modelshould have a quality metric above a given threshold, e.g. 90%. Otherrules can be related to explainability, data drift, etc. Rules forvalidating models are known to one of skill in the art and not explainedin more detail herein. The set of rules to be applied can be selected orcreated by a user, such as Chief Risk Officer, and can vary in differentembodiments depending on the purpose for which the models are beingbuilt. The set of rules are encoded in the AI governance policy engineimplemented by the control unit 108 such that the AI governance policyengine is able to validate a model against these rules. For example, theAI governance policy engine can check if the model was built usinggoverned data or not. One example technique for performing this check ongoverned data can be performed by comparing a list of approved governeddata catalogs to metadata information on the data used in the model thatis captured when the model is built. For example, a data scientist camspecify the training data used to build the model. This can be a dataasset which is present in the project in which the model is being built.Thus, model development tools maintain a concept of projects whichcontain all the artefacts which are used to build the model. Data assetsin the project can be copied from a governed catalog. The AI governancepolicy engine checks that the data asset used to build the model hasbeen copied from a governed catalog.

In addition, the control unit 108 is configured, in some embodiments, toautomatically evaluate a model against the set of rules in response todetecting that the model has been stored in the repository 106. Afterbuilding a model, a data scientist typically uploads the model to arepository, such as repository 106. The control unit 108 can beconfigured, in some embodiments, to periodically check for modelsuploaded to the repository since a last check and to analyze any newlydiscovered models against the rules and policies 112. In otherembodiments, uploading a model to the repository 106 can automaticallytrigger a command sent to the control unit 108 to analyze the newlyuploaded model against the rules and policies 112. If the AI governancepolicy engine detects any rule or policy violations, a notification issent to one or more of the client devices 102 to notify the user of theviolation. In this way, the user can correct or update the model tocomply with the rules. This reduces the likelihood that the model willbe rejected when a human validator reviews the model, thus saving timein the model development cycle.

Furthermore, the embodiments described herein ensure that the model isevaluated, and the model metrics are computed, in a tamper proof manner.If a user is allowed to compute and directly update the metrics, it ispossible to set a value such that the model will pass the automatedvalidation checks, e.g., accuracy to 99% and fairness to 95%, withoutactually having done any computation. Hence, there is a need for asystem which will compute these metrics and directly update it in arepository where it is not manually editable by users. Only authorizedtools are able to update this information and this information is usedby the AI governance policy engine to validate the model complies withthe AI governance rules and policies.

For example, in some embodiments, the repository 106 is a data catalog,such as, but not limited to IBM® Watson Knowledge Catalog, which cancapture model metadata. One or more parts of the model metadata aredefined by the repository 106 as non-editable by end users. Rather, thismetadata is defined as only editable by in-built model metricscomputation tools. One example of such a model metrics computation toolthat includes, but is not limited to, a system such as OpenScale. Forexample, a field in the model metadata can be included to identify whichelements can only be edited by a particular tool. The field can includethe service identification (ID) associated with a computation toolauthorized to make the edits. The service IDs can be registered with therepository 106 and only service IDs registered with the repository 106can make edits to the metadata defined as being non-editable by endusers. The computation tool is configured to compute these metrics anduse a unique service ID registered with the repository 106 to update themodel metrics in the model asset stored in the data catalog. Inoperation, once a model is built, it is saved to the repository 106 anda computation tool calculates metrics to validate the model. Aftercomputing the metrics, the computation tool can update metadataindicating values for the computed metrics. The AI governance policyengine can then verify if the computed metrics comply with the set ofrules corresponding to model metrics. In this way, an end user is notable to erroneously enter inaccurate metric data. Thus, the system 100is able to compute more accurate metrics and, consequently, moreaccurately determine if the model complies with the defined rules andpolicies 112.

In addition to the model metrics, system 100 can utilize other modelattributes in the AI governance policy engine to analyze a model. Onesuch example is the training data used to build the model. A datascientist could make use of non-governed data to build and train themodel, but the model could be erroneously labelled as having been builtusing governed data. In order to avoid such scenarios, in someembodiments, the AI governance policy engine first checks if there is apolicy that states that the model should only be built using governeddata when the AI governance policy engine is asked to validate themodel. If there is such a policy, the AI governance model checks if thetraining data pointed to by the label applied to the model is actuallyfrom a governed catalog. If yes, the AI governance policy engine readsthe data pointed by the model being validated and causes a newchallenger model to be built with the data. For example, the challengermodel can be built using various systems such as, but not limited to,AutoAI in IBM Watson® Studio. Such systems attempt to find the bestmodel that can be built using the training data. Metrics are thencomputed for the built challenger model. The AI governance policy enginethen compares the model metrics for the challenger model with computedmodel metrics for the model being validated. In some such embodiments,the AI governance policy engine determines that the model beingvalidated was built using non-governed data if the metrics for the modelbeing validated are worse than the metrics of the challenger model. Thisdecision is based on the assumption that if the model being validatedwere built using non-governed data then the typical problem with thatdata is that it has bad quality as it has not undergone proper dataquality checks. Hence the quality of a model built using such data willtypically be poor. Thus, if the quality of the model being validated isbetter than the one built using an AI system, it is assumed that themodel was built using the data pointed to by the label applied to themodel.

At periodic intervals, the model repository 106 checks with the AIgovernance policy engine implemented by the control unit 108 todetermine if the model in the repository conforms to the AI governancerules and policies. For example, the repository 106 can make a call tothe AI governance policy engine and send metadata about the model to theAI governance policy engine. As discussed above, in some embodiments,the metadata is tamper proof and updated only by authorized systemshaving a registered service ID. The AI governance policy engine checksif the model metrics conform to the policies and rules. If they do, thenthe model is flagged as one that passes the governance checks. If, onthe other hand, the model does not pass the governance checks, then theAI governance policy engine flags the model as having failed thevalidation. The AI governance policy engine can automatically output anotification to a corresponding client device associated with the model(e.g. a client device of the listed data scientist for the model)regarding the validation status of the model.

Additionally, the periodic checks enable the AI governance policy engineto detect a change in the policies or rules and update the validationstatus of the model. For example, when a model is pushed to therepository 106, the AI governance policy engine validates the modelagainst the rules and policies 112. The model may be found to complywith the rules and policies 112 at the time it is pushed to therepository 106. However, subsequent to being analyzed by the AIgovernance policy engine, the rules or policies can be changed by auser, such as the Chief Risk Officer. Due to the change in rules orpolicies, the model is no longer compliant. By performing periodicchecks, in some embodiments, the AI governance policy engine is able todetect the change in validation status of the model and send anotification to the user or data scientist that pushed the model to therepository 106 regarding the change. Additionally, in some embodiments,the AI governance policy engine is configured to detect changes to therules and policies 112 and, in response to such detected changes,automatically initiate a review of models that have been previousevaluated by the AI governance policy engine. Through these automatedvalidation checks, the system 100 helps ensure that the data scientisthas a chance to fix the model earlier in the development cycle ratherthan waiting for a human model validator to validate the model beforebeing notified of elements to be fixed. Thus, the delay in thedevelopment cycle of the model can be reduced.

In addition to the above, the AI governance policy engine also providesan application programming interface (API) using which enables a user tocheck on demand if the model meets the governance policies or if themodel needs to be improved so that it passes the validation check.Hence, the quality of models that are built is improved andproblems/issues in models related to violation of governance policieswill be known early in the lifecycle, thereby, reducing the time it willtake to deploy models to production.

FIG. 2 is a high-level block diagram of one embodiment of an examplecontrol unit 200 configured to implement an AI governance policy engine.In the example shown in FIG. 2 , the control unit 200 includes a memory225, storage 230, an interconnect (e.g., BUS) 220, one or moreprocessors 205 (also referred to as CPU 205 herein), an Input/Output(I/O) device interface 250, and a network interface 215. It is to beunderstood that the control unit 200 is provided by way of example onlyand that the control unit 200 can be implemented differently in otherembodiments. For example, in other embodiments, some of the componentsshown in FIG. 2 can be omitted and/or other components can be included.

Each CPU 205 retrieves and executes programming instructions stored inthe memory 225 and/or storage 230. The interconnect 220 is used to movedata, such as programming instructions, between the CPU 205, storage230, network interface 215, and memory 225. The interconnect 220 can beimplemented using one or more busses. The CPUs 205 can be a single CPU,multiple CPUs, or a single CPU having multiple processing cores invarious embodiments. In some embodiments, a processor 205 can be adigital signal processor (DSP). Memory 225 is generally included to berepresentative of a random access memory (e.g., static random accessmemory (SRAM), dynamic random access memory (DRAM), or Flash). Thestorage 230 is generally included to be representative of a non-volatilememory, such as a hard disk drive, solid state device (SSD), removablememory cards, optical storage, or flash memory devices. In analternative embodiment, the storage 230 can be replaced by storagearea-network (SAN) devices, the cloud, or other devices connected to thecontrol unit 200 via a communication network coupled to the networkinterface 215.

In some embodiments, the memory 225 stores policy engine instructions210 and the storage 230 stores governance rules and policies data 209.This governance rules and policies data 209 can include rules such asdiscussed above. In other embodiments, the instructions 210 and thegovernance rules and policies data 209 are stored partially in memory225 and partially in storage 230, or they are stored entirely in memory225 or entirely in storage 230, or they are accessed over a network viathe network interface 215. Additionally, as discussed above, thegovernance rules and policies data 209 can be stored in a database ormemory device accessed via the network interface 215, such as repository106, rather than being locally attached or integrated with the controlunit 200.

When executed, the instructions 210 cause the CPU 205 to analyze models,such as models 114 received over the network interface 215, to performthe functionality discussed above. Additionally, the control unit 200can receive user instructions requesting validation of a specified modelvia either the network interface 215 or the I/O device interface 250.The instructions 210 further cause the CPU 205 to output signals andcommands to a device via network interface 215 and/or I/O deviceinterface 250 to indicate if the model complies with the governancerules and policies.

FIG. 3 depicts one embodiment of an example method 300 of enforcinggovernance policy using tamper proof model metrics. The method 300 canbe implemented by a control unit, such as control unit 108. For example,the method 300 can be implemented by a CPU, such as CPU 205 in controlunit 200, executing instructions, such as policy engine instructions210. It is to be understood that the order of actions in example method300 is provided for purposes of explanation and that the method can beperformed in a different order in other embodiments. Similarly, it is tobe understood that some actions can be omitted or additional actions canbe included in other embodiments.

At 302, a model stored in a repository, such as repository 106, isidentified as a model to be validated. For example, in some embodiments,a model to be validated is identified based on user instructionsreceived via an application programming interface (API). The userinstructions can include a command or request to initiate validation andspecify the model to be validated. In other embodiments, identifying themodel to be validated can include detecting that a model has been pushedto the repository. Thus, in such embodiments, the validation isinitiated automatically in response to detecting that the model has beenpushed to the repository. In other embodiments, identifying the model ispart of a periodic validation check. For example, the control unit 108can periodically perform recurring validation checks on models stored inthe repository 106 according to a schedule. In this way, changes torules and/or policies after an initial validation check can be capturedand used to re-evaluate the models stored in the repository 106. In someembodiments, the control unit 108 is configured to detect changes to anencoded rule and/or policy and, in response to detecting such a change,the control unit 108 is configured to identify models stored in therepositor 106 which have not been evaluated since the detection of thechange. Those models which have not been evaluated after the detectedchange, are then identified as models to be validated.

At 304, one or more model metrics for the model to be validated areautomatically computed and recorded in a tamper-proof manner, asdiscussed above. For example, in some embodiments, a service ID of acomputation tool authorized to edit metadata of the model is registeredwith the repository. The control unit 108 and/or the repository 106 thenprevents or prohibits edits to the metadata of the model by end usersand any tool without a service ID registered with the repository as anauthorized tool. In this way, the model metrics can be computed only byauthorized tools and those metrics can only be stored or edited by theauthorized tools. This enables the metrics to be computed in atamper-proof manner which increases the accuracy of computations indetermining compliance of a model with the rules and policies.

At 306, the computed tamper-proof metrics are compared by the controlunit 108 with one or more encoded rules and policies to determine if themodel to be validated complies with the one or more encoded rules andpolicies. As discussed above, since the model metrics are computedautomatically and in a tamper-proof manner, the control unit 108 is ableto more accurately determine whether the model complies with the rulesand policies since the model metrics are not subject to human error orunauthorized editing. Additionally, in some embodiments, comparing thetamper-proof metrics also includes verifying metadata entered by a humanuser and comparing the verified metadata to one or more rules/policies.For example, as described above, a user can specify which training datawas used to build the model. However, the data specified by the usercould be erroneously specified. Since some rules include only usingtraining data from approved governed data sources, the control unit 108can verify that that the specified training data is the data actuallyused in building the model.

For example, in some embodiments, to verify the training data used, thecontrol unit 108 retrieves the training data specified or identified ina label or metadata associated with the model being validated. Thecontrol unit 108 then automatically builds a sample model using theretrieved training data. For example, the control unit 108 can utilizetools such as AutoAI in IBM Watson® Studio to build the sample model.However, it is to be understood that other tools for automaticallybuilding the sample model can be used in other embodiments. The controlunit 108 then computes one or more metrics for the sample model andcompares the one or more metrics for the sample model with the one ormore metrics for the model to be validated. Based on the comparison ofthe metrics, the control unit 108 is able to verify whether the model tobe validated was built using the identified training data. For example,as discussed above, if the metrics for the sample model are better orcomply more with the rules and policies than the metrics for the modelto be validated, then the control unit 108 can determine that the modelto be validated was not built using the training data.

At 308, a notification is output to a device indicating a validationstatus of the model to be validated based on the comparison of thecomputed tamper-proof metrics with the one or more encoded rules andpolicies. For example, this notification can be in response to anon-demand request for validation, such as through the API, in responseto a periodic validation check, in response to a detected change in therules/policies and/or in response to an initial validation check. Thenotification can include a high-level status such as whether the modelcomplies with the rules and policies. Additionally, it can includespecific values for computed metrics and/or an indication of whichrules/policies are not complied with. The notification can be a textmessage, email, chat message, etc. By automatically providing thenotification to a device of a user or data scientist associated with themodel, the user or data scientist is able to correct or fix the modelwithout having to wait for a human validator to review the model as inconventional development cycles.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be accomplished as one step, executed concurrently,substantially concurrently, in a partially or wholly temporallyoverlapping manner, or the blocks may sometimes be executed in thereverse order, depending upon the functionality involved. It will alsobe noted that each block of the block diagrams and/or flowchartillustration, and combinations of blocks in the block diagrams and/orflowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts orcarry out combinations of special purpose hardware and computerinstructions.

In addition, one or more functionalities of the system 100 can beimplemented in a cloud computing environment. However, it is to beunderstood that although this disclosure includes a detailed descriptionon cloud computing, implementation of the teachings recited herein arenot limited to a cloud computing environment. Rather, embodiments of thepresent invention are capable of being implemented in conjunction withany other type of computing environment now known or later developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 4 , illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 includes one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer device 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 4 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 5 , a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 4 ) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 5 are intended to be illustrative only and embodiments ofthe invention are not limited thereto. As depicted, the following layersand corresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and automatic model validation 96.

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat any arrangement, which is calculated to achieve the same purpose,may be substituted for the specific embodiments shown. Therefore, it ismanifestly intended that this invention be limited only by the claimsand the equivalents thereof.

What is claimed is:
 1. A system comprising: an interface; a memoryconfigured to store governance rules and policies; and a processorcommunicatively coupled to the memory and to the interface, wherein theprocessor is configured to: perform an identifying operation selectedfrom the group consisting of: a) identifying a model to be validatedthat is stored in a repository responsive to detecting that the modelwas pushed to the repository, and b) identifying the model to bevalidated comprises: detecting a change in the one or more encoded rulesand policies; and in response to detecting the change, identifying themodel to be validated based on the model having not been validatedsubsequent to the detected change; automatically compute and record oneor more model metrics for the model to be validated in a tamper-proofmanner; compare the computed tamper-proof metrics with one or moreencoded rules and policies to determine if the model to be validatedcomplies with the one or more encoded rules and policies; and output anotification to a device indicating a validation status of the model tobe validated based on the comparison of the computed tamper-proofmetrics with the one or more encoded rules and policies.
 2. The systemof claim 1, wherein identifying the model to be validated comprises:receipt of instructions via an application programming interface (API)to initiate validation of the model.
 3. The system of claim 1, whereinidentifying the model to be validated comprises: identification of themodel to be validated in response to detecting that the model was pushedto the repository.
 4. The system of claim 1, wherein identification ofthe model to be validated comprises configuring the processor to:periodically initiate a validation check on the model.
 5. The system ofclaim 1, wherein identifying the model to be validated comprises:detection of a change in the one or more encoded rules and policies; andin response to the detection of the change, identifying the model to bevalidated based on the model having not been validated subsequent to thedetected change.
 6. The system of claim 1, wherein automaticallycomputing and recording one or more model metrics for the model in atamper-proof manner comprises configuring the processor to: registerwith the repository a service identification (ID) of a computation toolauthorized to edit metadata of the model to be validated; and prohibitedits to the metadata of the model to be validated that are notperformed by the computation tool having the registered service ID. 7.The system of claim 1, wherein comparing the computed tamper-proofmetrics with one or more encoded rules and policies to determine if themodel complies with the one or more encoded rules and policies includesconfiguring the processor to: identify training data specified in alabel of the model to be validated that indicates the model to bevalidated was built with the training data; retrieve the identifiedtraining data; automatically build a sample model using the retrievedtraining data; automatically compute one or more metrics for the samplemodel; compare the one or more metrics for the sample model with the oneor more for the model to be validated; and verify whether the model tobe validated was built using the identified training data based on thecomparison.
 8. A system comprising: an interface; a memory configured tostore governance rules and policies; and a processor communicativelycoupled to the memory and to the interface, wherein the processor isconfigured to: identify a model to be validated that is stored in arepository; automatically compute and record one or more model metricsfor the model to be validated in a tamper-proof manner; compare thecomputed tamper-proof metrics with one or more encoded rules andpolicies to determine if the model to be validated complies with the oneor more encoded rules and policies; and output a notification to adevice indicating a validation status of the model to be validated basedon the comparison of the computed tamper-proof metrics with the one ormore encoded rules and policies; wherein the automatic computation andrecordation of the one or more model metrics for the model in atamper-proof manner comprises configuring the processor to: registerwith the repository a service identification (ID) of a computation toolauthorized to edit metadata of the model to be validated; and prohibitedits to the metadata of the model to be validated that are notperformed by the computation tool having the registered service ID. 9.The system of claim 8, wherein the identification of the model to bevalidated comprises configuring the processor to: receive instructionsvia an application programming interface (API) to initiate validation ofthe model.
 10. The system of claim 8, wherein identification of themodel to be validated comprises configuring the processor to: identifythe model to be validated in response to the detection that the modelwas pushed to the repository.
 11. The system of claim 8, wherein theidentification of the model to be validated comprises configuring theprocessor to: periodically initiate a validation check on the model. 12.The system of claim 8, wherein the identification of the model to bevalidated comprises configuring the processor to: detect a change in theone or more encoded rules and policies; and in response to the detectionof the change, identify the model to be validated based on the modelhaving not been validated subsequent to the detected change.
 13. Thesystem of claim 8, wherein the comparison of the computed tamper-proofmetrics with one or more encoded rules and policies to determine if themodel complies with the one or more encoded rules and policies includesconfiguring the processor to: identify training data specified in alabel of the model to be validated that indicates the model to bevalidated was built with the training data; retrieve the identifiedtraining data; automatically build a sample model using the retrievedtraining data; automatically compute one or more metrics for the samplemodel; compare the one or more metrics for the sample model with the oneor more for the model to be validated; and verify whether the model tobe validated was built using the identified training data based on thecomparison.
 14. A system comprising: an interface; a memory configuredto store governance rules and policies; and a processor communicativelycoupled to the memory and to the interface, wherein the processor isconfigured to: identify a model to be validated that is stored in arepository; automatically compute and record one or more model metricsfor the model to be validated in a tamper-proof manner; compare thecomputed tamper-proof metrics with one or more encoded rules andpolicies to determine if the model to be validated complies with the oneor more encoded rules and policies; and output a notification to adevice indicating a validation status of the model to be validated basedon the comparison of the computed tamper-proof metrics with the one ormore encoded rules and policies; wherein the comparison of the computedtamper-proof metrics with one or more encoded rules and policies todetermine if the model complies with the one or more encoded rules andpolicies includes configuring the processor to: identify training dataspecified in a label of the model to be validated that indicates themodel to be validated was built with the training data; retrieve theidentified training data; automatically build a sample model using theretrieved training data; automatically compute one or more metrics forthe sample model; compare the one or more metrics for the sample modelwith the one or more for the model to be validated; and verify whetherthe model to be validated was built using the identified training databased on the comparison.
 15. The system of claim 14, wherein theidentification of the model to be validated comprises configuring theprocessor to: receive instructions via an application programminginterface (API) to initiate validation of the model.
 16. The system ofclaim 14, wherein the identification of the model to be validatedcomprises configuring the processor to: identify the model to bevalidated in response to detecting that the model was pushed to therepository.
 17. The system of claim 14, wherein the identification ofthe model to be validated comprises configuring the processor to:periodically initiate a validation check on the model.
 18. The system ofclaim 14, wherein the identification of the model to be validatedcomprises configuring the processor to: detect a change in the one ormore encoded rules and policies; and in response to the detection of thechange, identify the model to be validated based on the model having notbeen validated subsequent to the detected change.
 19. The system ofclaim 14, wherein the automatic computation and recordation of the oneor more model metrics for the model in a tamper-proof manner comprisesconfiguring the processor to: register with the repository a serviceidentification (ID) of a computation tool authorized to edit metadata ofthe model to be validated; and prohibit edits to the metadata of themodel to be validated that are not performed by the computation toolhaving the registered service ID.